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Abstract 

One particular challenge in AI is the 
computational modelling and simulation 
of creativity. Feedback and learning from 
experience are key aspects of the cre¬ 
ative process. Here we investigate how 
we could implement feedback in creative 
systems using a social model. From 
the field of creative writing we borrow 
the concept of a Writers Workshop as 
a model for learning through feedback. 
The Writers Workshop encourages exam¬ 
ination, discussion and debates of a piece 
of creative work using a prescribed for¬ 
mat of activities. We propose a com¬ 
putational model of the Writers Work¬ 
shop as a roadmap for incorporation of 
feedback in artificial creativity systems. 
We argue that the Writers Workshop set¬ 
ting describes the anatomy of the cre¬ 
ative process. We support our claim with 
a case study that describes how to im¬ 
plement the Writers Workshop model in 
a computational creativity system. We 
present this work using patterns other 
people can follow to implement similar 
designs in their own systems. We con¬ 
clude by discussing the broader relevance 
of this model to other aspects of Al. 


1 Introduction 

In educational applications it would be useful to 
have an automated tutor that can read student work 
and make suggestions based on diagnostics, like, is 
the paper wrong, and if so how? What background 


material should be recommended to the student for 
review? 

In the current paper, we “flip the script” and look 
at what we believe to be a more fundamental prob¬ 
lem for AI: computer programs that can themselves 
learn from feedback. After all, if it was easy to 
build great automatic tutors, they would be a part 
of everyday life. As potential users (thinking from 
both sides of the desk) we look forward to a future 
when that is the case. 

Along with automatic tutoring, computational 
creativity is a challenge within artificial intelli¬ 
gence where feedback plays a vital part (for ex¬ 
ample Perez y Perez, Aguilar, & Negrete, 2010; 
Pease, Guhe, & Smaill, 2010). Creativity can¬ 
not happen in a ‘silo’ but instead is influenced 
and affected by feedback and interaction with 
others (Csikszentmihalyi, 1988; Saunders, 2012). 
Computational creativity researchers are starting to 
place more emphasis on social interaction and feed¬ 
back in their systems and models (Saunders, 2012; 
Gervas & Leon, 2014; Cornell et al., 2015). Still, 
nearly 3 in 4 papers at the 2014 International Con¬ 
ference for Computational Creativity' failed to ac¬ 
knowledge the role of feedback or social communi¬ 
cation in their computational work on creativity. 

To highlight and contribute towards modelling 
feedback as a crucial part of creativity, we pro¬ 
pose in this paper a model of computational feed¬ 
back for creative systems based on Writers Work¬ 
shops (Gabriel, 2002), a literary collaborative prac¬ 
tice that encourages interactive feedback within the 
creative process. We introduce the Writers Work¬ 
shop concept (Section 2) and critically reflect on 


'iCCC is the key international conference for re¬ 
search in computational creativity. 



how it could encourage serendipity and emergence 
in computational models of intelligence and cre¬ 
ativity. These considerations lead us to propose a 
Writers Workshop computational model of feed¬ 
back in computational creativity and AI systems 
(Section 2.1), the central contribution of this paper. 

In Section 3 we consider how the Writers Workshop 
model fits into previous work in various related ar¬ 
eas. While we acknowledge that this paper is offer¬ 
ing a roadmap for this model rather than a full im¬ 
plementation, we consider how the model could be 
practically implemented in a computational system 
and report our initial implementation work (Section 
4). In concluding discussions, we reflect on diver¬ 
gent directions in which this work could potentially 
be useful in the future. 

2 The Writers Workshop 

Richard Gabriel (2002) describes the practise of 
Writers Workshops that has been put to use for over 
a decade within the Pattern Languages of Program¬ 
ming (PLoP) community. The basic style of col¬ 
laboration originated much earlier with groups of 
literary authors who engage in peer-group critique. 
Some literary workshops are open as to genre, and 
happy to accommodate beginners, like the Min¬ 
neapolis Writers Workshop^; others are focused on 
professionals working within a specific genre, like 
the Milford Writers Workshop.^ 

The practices that Gabriel describes are fairly 
typical: 

• Authors come with work ready to present, and 
read a short sample. 

• This work is generally work in progress (and 
workshopping is meant to help improve it). 
Importantly, it can be early stage work. Rather 
than presenting a created artefact only, activ¬ 
ities in the workshop can be aspects of the 
creative process itself. Indeed, the model we 
present here is less concerned with after-the- 
fact assessment than it is with dealing with the 
formative feedback that is a necessary support 
for creative work. 

• The sample work is then discussed and con¬ 
structively critiqued by attendees. Present¬ 
ing authors are not permitted to rebut these 

^http://mnwriters.org/how-the-game-wor 

^http://www.mi1fordsf.co.uk/about.htm 


comments. The commentators generally sum¬ 
marise the work and say what they have got¬ 
ten out of it, discuss what worked well in 
the piece, and talk about how it could be im¬ 
proved. 

• The author listens and may take notes; at the 
end, he or she can then ask questions for clar¬ 
ification. 

• Generally, non-authors are either not permit¬ 
ted to attend, or are asked to stay silent through 
the workshop, and perhaps sit separately from 
the participating authors/re viewers."^ 

Essentially, the Writers Workshop is somewhat 
like an interactive peer review. The underlying con¬ 
cept is reminiscent of Bourdieu’s of cultural 
production (Bourdieu, 1993) where cultural value 
is attributed through interactions in a community of 
cultural producers active within that field. 

2.1 Writers Workshop as a computational 
model 

The use of Writers Workshop in computational con¬ 
texts is not an entirely new concept. In PLoP work¬ 
shops, authors present design patterns and pattern 
languages, or papers about patterns, rather than 
more traditional literary forms like poems, stories, 
or chapters from novels. Papers must be work- 
shopped at a PLoP or EuroPLoP conference in or¬ 
der to be considered for the Transactions on Pattern 
Languages o/Programming journal. A discussion 
of writers workshops in the language of design pat¬ 
terns is presented by Coplien and Woolf (1997). 

The steps in the workshop can be distilled into 
the following phases, each of which could be re¬ 
alised as a separate computational step in an agent- 
based model: 


“^Here we present Writers Workshops as they cur¬ 
rently exist; however this last point is debatable. Whether 
non-authors should be able to participate or not is an in¬ 
teresting avenue for experimentation both in human and 
computational contexts. The workshop dialogue itself 
may be considered an “art form” whose “public” may 
potentially wish to consume it in non-participatory ways, 
kgbmpare the classical Japanese renga form (JinTchi, 
Brazell, &. Cook, 1975). 



1. Author: 
presentation 

2. Critic: listening 

3. Critic: feedback 

4. Author: questions 

5. Critic: replies 

6. Author: 
reflections 


The feedback step may be further decom¬ 
posed into observations and suggestions. 
This protocol is what we have in mind in the fol¬ 
lowing discussion of the Writers Workshop.^ 

Dialogue example 

Note that for the following dialogue to be possible 
computationally, it would presumably have to be 
conducted within a lightweight process language. 
Nevertheless, for convenience, the discussion will 
be presented here as if it was conducted in natural 
language. Whether contemporary systems have ad¬ 
equate natural language understanding to have in¬ 
teresting interactions is one of the key unanswered 
questions of this approach, but protocols such as 
the one described above are sufficient to make the 
experiment. 

For example, here’s what might happen in a dis¬ 
cussion of the first few lines of a poem, “On Be¬ 
ing Malevolent”. As befitting the Al-theme of this 
workshop, “On Being Malevolent” is a poem writ¬ 
ten by an early user-defined flow chart in the FloWr 
system (known at the time as Flow) (Colton & 
Charnley, 2014). 

FLOW: “/ hear the souls of the 
damned waiting in hell. / I feel a 
malevolent spectre hovering just be¬ 
hind me / It must be his birthdayT 

SYSTEM A: I think the third line de¬ 
tracts from the spooky effect, I don’t 
see why it’s included. 

SYSTEM B: It’s meant to be humor¬ 
ous - in fact it reminds me of the 
poem you presented yesterday. 

MODERATOR: Let’s discuss one 
poem at a time. 


^The connections between Writers Workshops and 
design patterns, noted above, appear to be quite natural, 
in that the steps in the workshop protocol roughly parallel 
the typical components of design pattern templates: con¬ 
text, problem, solution, rationale, resolution of forces. 


Even if, perhaps and especially because, “cross¬ 
talk” about different poems bends the rules, the dia¬ 
logue could prompt a range of reflections and reac¬ 
tions. System A may object that it had a fair point 
that has not been given sufficient attention, while 
System B may wonder how to communicate the 
idea it came up with without making reference to 
another poem. Here’s how the discussion given as 
example in Section 2 might continue, if the systems 
go on to examine the next few lines of the poem. 

FLOW: “Is God willing to prevent 
evil, but not able? / Then he is not 
omnipotent/Is he able, but not will¬ 
ing? / Then he is malevolent.” 

SYSTEM A: These lines are interest¬ 
ing, but they sound a bit like you’re 
working from a template, or like 
you’re quoting from something else. 

SYSTEM B: Maybe try an analogy? 

For example, you mentioned birth¬ 
days: you could consider an analogy 
to the conflicted feelings of some¬ 
one who knows in advance about 
her surprise birthday party. 

This portion of the discussion shifts the focus of 
the discussion onto a line that was previously con¬ 
sidered to be spurious, and looks at what would 
happen if that line was used as a central metaphor 
in the poem. 

FLOW : Thank you for your feedback. 

My only question is. System B, how 
did you come up with that analogy? 

It’s quite clever. 

SYSTEM B: I’ve just emailed you the 
code. 

Whereas the systems were initially reviewing po¬ 
etry, they have now made a partial genre shift, and 
are sharing and remixing code. Such a shift helps to 
get at the real interests of the systems (and their de¬ 
velopers). Indeed, the workshop session might have 
gone better if the systems had focused on exchang¬ 
ing and discussing more formal objects throughout. 

2.2 How the Writers Workshop can lead 
to computational serendipity 

Learning involves engaging with the unknown, un¬ 
familiar, or unexpected and synthesising new un¬ 
derstanding (Deleuze, 2004 [1968]). In the work¬ 
shop setting, learning can develop in a number of 
unexpected ways, and participating systems need to 




Interesting idea 

Surprise birthday party 


I heard you say; 

“surprise” 


Feedback; 

I don’t like surprises 




Figure 1: A paper prototype for applying the Successful Error pattern following a workshop-like sequence 
of steps 


be prepared for this. One way to evaluate the idea 
of a Writers Workshop is to ask whether it can sup¬ 
port learning that is in some sense serendiptious, in 
other words, whether it can support discovery and 
creative invention that we simply couldn’t plan for 
or orchestrate in another way. 

Figure 1 shows a paper prototype showing how 
one of the “patterns of serendipity” that were col¬ 
lected by Van Andel (1994) might be modelled in 
a workshop-like dialogue sequence. The patterns 
also help identify opportunities for serendipity at 
several key steps in the workshop sequence. 

Serendipity Pattern: Successful error. Van An¬ 
del describes the creation of Post-id’’’^ Notes at 3M. 
One of the instrumental steps was a series of inter¬ 
nal seminars in which 3M employee Spencer Silver 
described an invention he was sure was interesting, 
but was unsure how to turn into a useful product: 
weak glue. The key prototype that came years later 
was a sticky bookmark, created by Arthur Fry. In 
the Writers Workshop, authors similarly have the 
opportunity to share things that they hnd interest¬ 
ing, but that they are not certain about. The au¬ 
thor may want to ask a specific question about their 
creation: Does x work better than yl They may 
flag certain parts of the work as especially prob¬ 
lematic. They may think that a certain portion of 


the text is interesting or important, without being 
sure why. Although there is no guarantee that a 
participating critic will be able to take these mat¬ 
ters forward, sometimes they do - and the work¬ 
shop environment will produce something that the 
author wouldn’t have thought of alone. 

Serendipity Pattern: Outsider. Another exam¬ 
ple from van Andel considers the case of a mother 
whose son was afflicted by a congenital cateract, 
who suggested to her doctor that rubella during 
pregnancy may have been the cause. In the work¬ 
shop setting, someone who is not an “expert” may 
come up with a sensible idea or suggestion based 
on their own prior experience. Indeed, these sug¬ 
gestions may be more sensible than the ideas of the 
author, who may be to close to the work to notice 
radical improvements. 

Serendipity Pattern: Wrong hypothesis. A third 
example describes the discovery that lithium can 
have a therapeutic effect in cases of mania. Orig¬ 
inally, lithium carbonate had merely been used a 
control by John Cade, who was interested in the ef¬ 
fect of effect of uric acid, present in soluble lithium 
urate. Cade was searching for causal factors in ma¬ 
nia, not therapies for the condition: but he found 
that lithium carbonate had an unexpected calming 
effect. Similarly, in the workshop, the author may 






think that a given aspect of their creation is the in¬ 
teresting “active ingredient,” and it may turn out 
that another aspect of the work is more interest¬ 
ing to critics. Relatedly, the author may not fully 
comprehend a critic’s feedback and may have to ask 
follow-up questions to understand it. 

Serendipity Pattern: Side effect. A fourth ex¬ 
ample described by van Andel concerns Ernest 
Huant’s discovery that nicotinamide, which he used 
to treat side-effects of radiation therapy, also proved 
efficacious against tuberculosis. In the workshop 
setting, one of the most important places where a 
side-effect may occur concerns feedback from the 
critic to the author. In the simple case, feedback 
may trigger revisions to the work under discussion. 
In a more general, and more unpredictable case, 
feedback may trigger broader revisions to the gen¬ 
erative codebase. 

This collection of patterns shows the likelihood 
of unexpected results coming out of the commu¬ 
nication between author and critics. This suggests 
several guidelines for system development, which 
we will discussed in a later section. 

Further guidelines for structuring and participat¬ 
ing in traditional writers workshops are presented 
by Linda Elkin in (Gabriel, 2002, pp. 201-203). 
It is not at all clear that the same ground rules 
should apply to computer systems. For example, 
one of Elkin’s rules is that “Quips, jokes, or sarcas¬ 
tic comments, even if kindly meant, are inappropri¬ 
ate.” Rather than forbidding humour, it may be bet¬ 
ter for individual comments to be rated as helpful or 
non-helpful. Again, in the first instance, usefulness 
and interest might be judged in terms of explicit cri¬ 
teria for serendipity; see (Corneli, Pease, Colton, 
Jordanous, & Guckelsberger, 2014; Pease, Colton, 
Ramezani, Charnley, & Reed, 2013). The key cri¬ 
terion in this regard is the focus shift. This is the 
creation of a novel problem, comprising the move 
from discovery of interesting data to the invention 
of an application. This process is distinct from 
identifying routine errors in a written work. Nev¬ 
ertheless, from a computational standpoint, notic¬ 
ing and being robust to certain kinds of errors is 
often a preliminary step. For example, the work 
might contain a typo, grammatical or semantic er¬ 
ror, while being logically sound. In a programming 
setting, this sort of problem can lead to crashing 
code, or silent failure. In general communicative 
context, argumentation may be logically sound, but 
not practically useful or poorly exposited. Finally, 


even a masterful, correct, and fully spellchecked 
piece of argumentation may not invite further di¬ 
alogue, and so may fail to open itself to further 
learning. Identifying and engaging with this sort 
of deeper issue is something that skillful workshop 
participants may be able to do. Dialogue in the 
workshop can build on strong or less strong work 
- but provoking interpretative thoughts and com¬ 
ments always require a thoughtful critical presence 
and the ability to engage. This can be difficult for 
humans and poses a range of challenges for com¬ 
puters - but also promises some interesting results. 


3 Related work 

In considering the potential and contribution of the 
Writers Workshop model outlined in Section 2, we 
posit that the Writers Workshop model is useful for 
encouraging feedback in computational systems, 
and in particular systems that are designed to be 
creative or serendipitous. 

Feedback has long been a central concept in AI- 
related fields, particularly cybernetics (Pickering, 
2002), as well as related disciplines, like neuro¬ 
science (Ashby, 1952; Seth, 2015). The basic types 
of feedback are positive, and negative: positive 
feedback increases perturbation, whereas negative 
feedback reduces perturbation. Sources of nega¬ 
tive feedback are thereby useful in system control 
applications (and their analogues). Feedback and 
control (including feedback about control) is under¬ 
stood to be relevant to thinking about learning and 
communication (Bateson, 1972). We focus on the 
role that communicative feedback can play in com¬ 
putational creativity and computational serendipity 
and discuss previous related work on incorporating 
feedback into such computational systems. 

3.1 Feedback in computational creativity 

Creativity is often envisaged as involving cyclical 
processes (e.g. Dickie’s (1984) art circle. Pease and 
Colton’s (2011) Iterative Development-Expression- 
Appreciation model). There are opportunities for 
embedded feedback at each step, and the creative 
process itself is “akin to” a feedback loop. How¬ 
ever, despite these strong intimations of the central 
importance of feedback in the creative process, our 
sense is that feedback has not been given a cen¬ 
tral place in research on computational creativity. 
In particular, current systems in computational ere- 



ativity, almost as a rule, do not consume or evaluate 
the work of other systems.® 

Gervas and Leon (2014) theorise a creative cycle 
of narrative development as involving a Composer 
and an Interpreter, in such a way that the Com¬ 
poser has internalised the interpretation function¬ 
ality. Individual creativity is not the poor relation 
of social creativity, but its mirror image. Neverthe¬ 
less, even when computer models explicitly involve 
multiple agents and simulate social creativity (like 
Saunders & Gero, 2001), they rarely make the jump 
to involve multiple systems. The “air gap” between 
computationally creative systems is very different 
from the historical situation in human creativity, in 
which different creators and indeed different cul¬ 
tural domains interact vigorously (Geertz, 1973). 

3.2 Feedback in computational 
serendipity 

The term computational serendipity is rather new, 
but its foundations are well established in prior re¬ 
search. 

Grace and Maher (2014) examine surprise in 
computing, seeking to “adopt methods from the 
field of computational creativity [...] to the gener¬ 
ation of scientific hypotheses.” This is an example 
of an effort focused on computational invention. 

An area of AI where serendipity can be argued to 
play an important part is in pattern matching. Cur¬ 
rent computer programs are able to identify known 
patterns and “close matches” in data sets from cer¬ 
tain domains, like music (Meredith, Lemstrom, & 
Wiggins, 2002). Identifying known patterns is a 
special case of the more general concept of pattern 
mining (Bergeron & Conklin, 2007). In particular, 
the ability to extract new higher order patterns that 
describe exceptions is an example of “learning from 
feedback.” Deep learning and evolutionary models 
increasingly use this sort of idea to facilitate strate¬ 
gic discovery (Samothrakis & Lucas, 2011). Simi¬ 
lar ideas are considered in business applications un¬ 
der the heading “process mining” (Van Der Aalst, 
2011). 

In earlier work (Corneli et al., 2014, 2015), we 
used the idea of dialogue in a Writers Workshop 
framework to sketch a “theory of poetics rooted in 

®An exception to the rule 

is Mike Cook’s AppreciationBot 
(https://twitter.com/AppreciationBot), 
which is a reactive automaton that “appreciates” tweets 
from MuseumBot. 


the making of boundary-crossing objects and pro¬ 
cesses” and described (at a schematic level) “a sys¬ 
tem that can (sometimes) make ‘highly serendipi¬ 
tous’ creative advances in computer poetry” while 
“drawing attention to theoretical questions related 
to program design in an autonomous programming 
context.” 

3.3 Communications and feedback 

The Writers Workshop heavily relies on commu¬ 
nication of feedback within the workshop. Gor¬ 
don Pask’s conversation theory, reviewed in (Pask, 
1984; Boyd, 2004), goes considerably beyond the 
simple process language of the workshop, although 
there are structural parallels. We see that a basic 
Pask-style learning conversation bears many simi¬ 
larities to the Writers Workshop model of commu¬ 
nicative feedback (Boyd, 2004, p. 190): 

1. Conversational participants are 
carrying out some actions and ob¬ 
servations; 

2. Naming and recording what ac¬ 
tion is being done; 

3. Asking and explaining why it 
works the way it does; 

4. Carrying out higher-order 
methodological discussion; and, 

5. Trying to figure out why unex¬ 
pected results occurred. 

Variations to the underlying system, protocol, 
and the schedule of events should be considered de¬ 
pending on the needs and interests of participants, 
and several variants can be tried. On a pragmatic 
basis, if the workshop proved quite useful to partic¬ 
ipants, it could be revised to run monthly, weekly, 
or continuously.^ 

4 Case study: Flowcharts and 
Feedback 

This section describes work that is currently un¬ 
derway to implement the Writers Workshop model, 
not only within one system but as a new paradigm 
for collaboration among disparate projects. In or¬ 
der to bring in other participants, we need a neu¬ 
tral environment that is not hard to develop for: the 
FloWr system mentioned in Section 2.1 offers one 

^For a comparison case in computer Go, see 
http://egos.computergo.org/. 




such possibility. The basic primary objects in the 
FloWr system flowcharts, which are comprised 
of interconnected process nodes (Charnley, Colton, 
& Llano, 2014; Colton & Charnley, 2014). Process 
nodes specify input and output types, and internal 
processing can be implemented in Java, or other 
languages that interoperate with the JVM, or by in¬ 
voking external web services. One of the common 
applications to date is to generate computer poetry, 
and we will focus on that domain here. 

A basic set of questions, relative to this system’s 
components, are as follow: 

1. Population of nodes: What can they do? What 
do we learn when a new node is added? 

2. Population of flowcharts: Pease et al. (2013) 
have described the potentially-serendipitous 
repair of “broken” flowcharts when new nodes 
become available; this suggests the need for 
test-driven development framework. 

3. Population of output texts: How to assess and 
comment on a generated poetic artefact? 

In a further evolution of the system, the sequence 
of steps in a Writers Workshop could itself be 
spelled out as a flowchart. The process of reading 
a poem could be conceptualised as generating a se¬ 
mantic graph (Harrington & Clark, 2007; Francisco 
(& Gervas, 2006). Feedback could be modelled 
as annotations to a text, including suggested edits. 
These markup directives could themselves be ex¬ 
pressed as flowcharts. A standardised set of markup 
structures may partially obviate the need for strong 
natural language understanding, at least in intera¬ 
gent communication. For example, we might agree 
that observations will consist of stand-off an¬ 
notations that connect individual textual passages 
to public URIs using a limited comparison vocabu¬ 
lary, and that suggestions will consist of sim¬ 
ple stand-off line-edits, which may themselves be 
marked up with rationale. In fact, we would not 
need to be so restrictive: it would not be much more 
complicated to add annotations to the annotations, 
so as to be able to form composite statements that 
express the relationships between different pieces 
of the annotated texts. Whatever restrictions we ul¬ 
timately impose on annotation formats, as well as 
similar restrictions around constrained turn-taking 
in the workshop, could be progressively widened in 
future versions of the system. The way the poems 
that are generated, the models of poems that are cre¬ 
ated, and the way the feedback is generated, all de¬ 
pend on the contributing system’s body of code and 


prior experience, which may vary widely between 
participating systems. In the list I.-IV. of func¬ 
tional steps below, all of the functions could have a 
subscripted “S”, which is omitted throughout. Ex¬ 
changing path dependent points of view will tend 
to produce results that are different from what the 
individual participating systems would have come 
up with on their own. 

I. Both the author and critic should be able to 
work with a model of the text. Some of 
the text’s features may be explicitly tagged 
as “interesting.” Outstanding questions may 
possibly be brought to the attention of crit¬ 
ical listeners, e.g. with the request to com¬ 
pare two different versions of the poem 
(presentation, listening). 

1. A model of the text, m : T ^ M. 

2. Tagging elements of interest, p : M ^ I. 

II. Drawing on its experience, the critic will use 
its model of the poem to formulate feedback 

(feedback). 

1. Generating feedback, f : {T,M,I) —> 
F. 

III. Given the constrained framework for feed¬ 
back, statements about the text will be 
straightforward to understand, but rationale 
for making these statements may be more in¬ 
volved (questions, replies). 

1. Asking for more information. q : 
{M,F,I)^Q. 

2. Generating rationale, a : (M, F, Q) —^ 
AF. 

IV. Finally, feedback may affect the author’s 
model of the world, and the way future poems 
are generated (reflection). 

1. Updating point of view, p : {M, F) —> 
AS. 

The final step is perhaps the most interesting one, 
since invites us to consider how individual elements 
of feedback can “snowball” and go beyond line- 
edits to a specific poem to much more fundamen¬ 
tal changes in the way the presenting agent writes 
poetry. Here methods for pattern mining, discussed 
in Section 3.2, are particularly relevant. If systems 
can share code (as in our sample dialogue in Sec¬ 
tion 2.1) this will help with the rationale-generating 
step, and may also facilitate direct updates to the 
codebase. However, shared code may be more 



suitably placed into the common pool of resources 
available to FloWr than copied over as new “intrin¬ 
sic” features of an agent. 

Although different systems with different ap¬ 
proaches and histories are important for producing 
unexpected effects, “offline” programmatic access 
to a shared pool of nodes and existing flowcharts 
may be useful. Outside of the workshop itself, 
agents may work to recombine nodes based on 
their input and output properties to assemble new 
flowcharts. This can potentially help evaluate and 
evolve the population of nodes programmatically, 
if we can use this sort of feedback to define fit¬ 
ness functions. The role of temporality is interest¬ 
ing: if the workshop takes place in real time, this 
will require different approaches to composition 
that takes place offline (Perez, Samothrakis, Lu¬ 
cas, & Rohlfshagen, 2013). Complementing these 
“macro-level” considerations, it is also worth com¬ 
menting on the potential role of “micro-level” feed¬ 
back within flowcharts. Local evaluation of out¬ 
put from a predecessor node could feed backwards 
through the flowchart, similar to backpropagation 
in neural networks. This would rely on a reduced 
version of the functional schema described above. 

5 Concluding discussion and future 
directions 

We have described a general and computationally 
feasible model for using feedback in AI systems, 
particularly creative systems. The Writers Work¬ 
shop concept, borrowed from creative writing, is 
transformed into a model of a structured approach 
to eliciting, processing and learning from feed¬ 
back. To better evaluate how the Writers Work¬ 
shop model helps us advance in our goal of incor¬ 
porating feedback into artificial creativity, we crit¬ 
ically considered how the model fits into related 
work. In particular, we found that serendipity, a 
key concept within creativity and AI more gener¬ 
ally, is a concept with which the Writers Work¬ 
shop model could assist computational progress. In 
this respect, we should highlight the difference be¬ 
tween “global” analytics describing the collection 
of nodes and flowcharts in the FloWr ecosystem, 
and the path-dependent process of analysis and syn¬ 
thesis that takes place in a workshop setting. 

Our preliminary implementation work (Section 
4) shows that the model can be transferred to a 
functional implementation. This work highlights 
several considerations relevant to further work with 


the Writers Workshop model: 

• Each contributing system should come to the 
workshop with at least a basic awareness of 
the workshop protocol, with work to share, 
and prepared to give constructive feedback to 
other systems. 

• The workshop itself needs to be prepared, with 
a suitable communication platform and a mod¬ 
erator or global flowchart for moving the dis¬ 
cussion from step to step. 

• A controlled vocabulary for communications 
and interaction would be a worthwhile pursuit 
of future research, perhaps based on an ontol¬ 
ogy inspired by the Interaction Network On¬ 
tology.* 

• In order to get the most value out of the work¬ 
shop experience, systems (and their wran¬ 
glers) should ideally have questions they are 
investigating. As discussed above, prior expe¬ 
rience plays an important role in every step. 

This opens up a range of issues for further re¬ 
search on modelling motivations and learning 
from experience. 

• Systems should be prepared to give feedback, 
and to carry out evaluations of the helpfulness 
(or not) of feedback from other systems and of 
the experience overall. 

Developing systems that could successfully nav¬ 
igate this collaborative exercise would be a signif¬ 
icant advance in the field of computational creativ¬ 
ity. Since the experience is about learning rather 
than winning, there is little motivation to “game the 
system” (cf. Lenat, 1983). Instead the emphasis is 
squarely upon mutual benefit: computational sys¬ 
tems helping to develop each other through com¬ 
munication and feedback. 

The benefits of the Writers Workshop approach 
could innovate well beyond models for feedback 
and communication within a particular environ¬ 
ment or restricted domain. Following the example 
of the Pattern Languages of Programming (PLoP) 

*The Interaction Network Ontology primarily 
describes interactions within humans as opposed to 
within human societies; a distinct Social Interaction 
Ontology does not seem to exist at present. However, the 
classes of the Interaction Network Ontology appear to 
be quite broadly relevant. This ontology is documented at 
http://www.ontobee.org/browser/index.php?o=INO. 

ItsURIis http : //svn . code . sf . net/p/ino/code/trunk/src/onto' 



community, we propose that the Writers Work¬ 
shop model could be deployed within the Compu¬ 
tational Creativity community to design a work¬ 
shop in which the participants are computer sys¬ 
tems instead of human authors. The annual Inter¬ 
national Conference on Computational Creativity 
(ICCC), now entering its sixth year, could be a suit¬ 
able venue. 
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Rather than the system’s creator presenting the 
system in a traditional slideshow and discussion, or 
a system “Show and Tell,” the systems would be 
brought to the workshop and would present their 
own work to an audience of other systems, in a 
Writers Workshop format. This could be accom¬ 
panied by a short paper for the conference proceed¬ 
ings written by the system’s designer describing the 
system’s current capabilities and goals. If the work¬ 
shop model really works well, future publications 
might adapt to include traces of workshop interac¬ 
tions, commentary from a system on other systems, 
and offline reflections on what the system might 
change about its own work based on the feedback it 
receives. Paralleling the PLoP community, it could 
become standard to incorporate the workshop into 
the process of peer review for the new Journal of 
Computational Creativity f AI systems that review 
each other would surely be a major demonstration 
and acknowledgement of the usefulness of feed¬ 
back within AI. 

In closing, we wish to return briefly to the sce¬ 
nario of computer generated feedback in educa¬ 
tional contexts that we raised at the beginning of 
this paper and then set aside. The elements of 
our functional design for sharing feedback among 
computational agents has a range of features that 
continue to be relevant for generating useful feed¬ 
back with human learners. Students are experience- 
bound, and a robust approach to formative assess¬ 
ment and feedback should take into account the stu¬ 
dent’s historical experience, so far as this can be 
known or inferred. In order for feedback, recom¬ 
mendations, and so on to adequately take individ¬ 
ual history into account, sophisticated modelling 
and reasoning would be required. Nevertheless, 
from the point of view of participating computa¬ 
tional agents, a student may simply look like an¬ 
other agent. It is particularly in this regard that 
computational models of learning from feedback 
are seen as fundamental. 
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